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ABSTRACT 


Digital image inpainting is a technique that restores an image by smoothly filling a hole in a purely automatic fashion. i. e. image inpainting fills the missing 
or damaged region in an image utilizing spatial information of its neighbouring region. The main goal of the Inpainting algorithm is to modify the damaged 
region in an image in such a way that the inpainted region is undetectable to the ordinary observers who are not familiar with the original image. Inpainting 
algorithm have numerous applications. It is helpfully used for restoration of old films and object removal in digital photographs. It is also applied to red-eye 
correction, super resolution, compression etc. This proposed work presents a brief comparative study of different image inpainting techniques like PDE 
based image inpainting, Exemplar based image inpainting, hybrid inpainting, and texture synthesis based image inpainting. 


Key-words: Inpainting, Object Removal, Exemplar, PDE based inpainting, Texture Synthesis 


Image Inpainting: 

Image inpainting refers to 
the process of filling in 
missing data ina 
designated region of the 
visual input, i.e. object of 
the process is to 
reconstruct missing parts 
or damaged image in such 
a way that the inpainted 
region cannot be detected 
by a casual observer. 
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1. INTRODUCTION 


Recentty, the Image Inpainting technology is a hotspot in graphics and that’s based on Applied Mathematics. Image 
inpainting is an iterative method for repairing damaged pictures or removing unnecessary elements from pictures. This activity 
consists of filling in the missing areas or modifying the damaged images in such a way that the inpainted region cannot be 
detected by a casual observer. It has important value in a heritage preservation, film and television special effects production, 
removing redundant objects etc. In the fine art museums, Inpainting is carried out by professional artist as shown in Figure 1, 
and usually its very time consuming process because it was the manual process. Inpainting technique has found widespread 
use in many applications such as restoration of old films and paintings, object removal in digital photos, compression, image 
coding, transmission, removal of occlusions, such as large unwanted regions, red eye correction, super resolution etc. Image 
inpainting reconstruct the damaged region or missing parts in an image utilizing spatial information of neighbouring region. In 
digital images, we only have the image we are working on available to us and thus we are filling in a hole that encompasses 
an entire object. It is impossible to replace that entire object based on the present’s information. So the aim of the inpainting 
algorithm is not only reconstruct what used to be in that hole, But instead to create a visually pleasing continuation of the data 
around the hole in such a way that it is not detectable by ordinary observer. 


1.1. Inpainting Problem 
Inpainting problem is defined in terms of Mathematical point of view, In a sequence say § , given only a subsequence of it, 


X estimate the whole § as S such that 1(s')= 1(X): where J denotes the information. Here is a simple example to explain it 


more clearly. Suppose there is a 
sequence {1,2,3,4, X,6} where X is 


the unknown element. If x is 
derived as 5, the whole sequence 
looks very “natural”, for 
1(X =5|{1,2,3,4,X,6})=0- Le, it 
takes the exact value as we 
expected. However, if X is derived 
as 10, then the whole sequence 
does tell us something unexpected, 
and 7(X =10|{1,2,3,4,X,6})>0- In 
case of inpainting, the generated 
plausible regions are commonly 
looks natural which indicates that 
no additional information can be 
reproduced out of nothing related. 


Figure 1 


Example of manual inpainting performed by a professional artist professional artist 
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Texture Synthesis: 
Texture synthesis based 
algorithms are used to 
complete the missing 
regions using similar 
neighbourhoods of the 
damaged pixels. The 
texture synthesis 
algorithms synthesize the 
new image pixels from an 
initial seed. And then 
strives to preserve the local 
structure of the image. 


Hybrid: 

Hybrid inpainting 
technique is also called as 
Image Completion. It is 
used for filling large target 
(missing) regions. And 
also preserves both 
structure and texture ina 
visually plausible manner. 
The hybrid approaches 
combine both texture 
synthesis and PDE based 
Inpainting for completing 
the holes. 


Application of Image 
Inpainting: 

Applications range from the 
reconstruction of missing 
blocks introduced by 
packet loss during wireless 
transmission, reversing of 
impairments, such as 
cracks, scratches, and dirt, 
in scanned photographs 
and digitized artwork 
images, to 
removal/introduction of 
image objects such as 
logos, stamped dates, text, 
persons, and special 
effects on the scene. 
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So we define general principle of Image inpainting. Principle of Inpainting: Let  c D stands for the region to be inpainted as 
shown in Figure 2, and @Q for its boundary. The objective is to fill the hole Q with appropriate gray values by interpolating the 
data located at the neighbourhood in the surrounding region @Q (e.g. boundaries of Q ). 

Most inpainting methods work as follows. As a first 
step the user manually selects the portions of the image 
that will be restored. Then image restoration is done 
automatically, by filling these regions in with new 
information coming from the surrounding area. In order to 
80 produce a perceptually plausible reconstruction, an 

inpainting technique must attempt to continue the isophote 

(line of equal gray value) as smoothly as possible inside 

the reconstruction region. In other words the missing 

region should be inpainted so that inpainted gray value 

and gradient extrapolate the gray value and gradient 

my outside the region. Several inpainting methods are based 

on the above ideas. The first category of inpainting is 

Diffusion Inpainting. In this approach missing region is 

; J filled by diffusing the image information from the known 

Figure 2 

Principle of Inpainting 


Image domain D 


region into the missing region at the pixel level. Basically 
these algorithms are based on theory of variational method 
and Partial Differential equation (PDE). The diffusion- 
based Inpainting algorithm produces superb results or 
filling the non-textured or relatively smaller missing region. The drawback of the diffusion process is it introduces some blur, 
which becomes noticeable when filling larger regions. All the PDE based inpainting models are more suitable for completing 
small, non-textured target region. The second category of Inpainting is exemplar-based Inpainting algorithm. This method of 
image Inpainting is an efficient approach to reconstructing large target regions. Exemplar-based Inpainting approach 
iteratively synthesizes the target region by most similar patch in the source region. These algorithms also overcome the 
drawbacks of PDE based inpainting. Also it removes smooth effect of the diffusion based Inpainting algorithm. This paper is 
organized as follows. Section-1 shows the Introduction of Image Inpainting problem. Section-2 describes the comparative 
study of different techniques for the image Inpainting which including exemplar based Inpainting, PDE based Inpainting, and 
Texture Synthesis based Inpainting, Hybrid Inpainting. Finally concluding remarks are given in Section-3. 


2. DIFFERENT TECHNIQUES OF IMAGE INPAINTING 


Nowadays, there are different techniques or approaches of image inpainting are available. So we can easily understand the 
methods and classify them into several categories as follows. |. Texture Synthesis based Inpainting. Il. PDE based Inpainting. 
lll. Exemplar based Inpainting. IV. Hybrid Inpainting. In this section we can briefly explain the concepts, usefulness, 
advantages, disadvantages, drawbacks, limitations of the Image Inpainting techniques. 


2.1. Texture Synthesis based Inpainting 

One of the earliest methods of image inpainting was to use general texture synthesis algorithms to complete the missing 
regions. The texture synthesis algorithms synthesize new image pixels from an initial seed and strive to preserve the local 
structures of the image. All the earlier Inpainting techniques utilized these methods to fill the missing region by sampling and 
copying pixels from the neighbouring area. Markov Random Field (MRF) is used to model the local distribution of a pixel and 
new texture is synthesized by querying existing texture and finding all similar neighbourhoods. Their differences lay mainly in 
how continuity is maintained between the inpainted hole and the existing pixels. These synthesis based techniques perform 
well only for a select set of images where completing the hole region with homogenous texture information would result in a 
natural completion. Then later, this technique was extended to fast synthesizing algorithm. This technique works by stitching 
together small patches of existing images referred to as image quilting. Bergen and Heeger, (1995) 
developed a parametric texture synthesis algorithm which can synthesize a matching texture, given a target texture. This was 
done by matching first order statistics of a linear filter bank which roughly match to the texture discrimination capabilities of 
Human Visual System [HVS]. Recently, a fast multi-resolution based image completion based and on texture analysis and 
synthesis was introduced fang et al., in 2000. In their method, the input image was analysed by a patch based method using 
principal component analysis (PCA) and a Vector Quantization (VQ) based techniques was used to speed up the matching 
process of the texture inside the hole region. 

Texture synthesis approaches (Efors et al.1999) can be categorized into three categories: Statistical (parametric), pixel- 
based and patch-based (non-parametric). Statistical methods are more likely to succeed in reproducing stochastic/irregular 
textures, but usually fail to reproduce structured/regular textures. On the other hand, pixel-based methods “build” on the 
sample texture pixel-by-pixel instead of applying filters on it, and their final outputs are of better quality than those of statistical 
methods, but they usually fail to grow large structured textures. Finally, patch-based methods “build” on a sample texture 
patch-by-patch as opposed to pixel-by-pixel, thus they yield faster and more plausible regular textures. A comparative study 
for patch-based texture synthesis algorithms has shown that “for handling special types of texture we have to develop the 
special purpose algorithms”. Taking this aim and the variety of algorithms for texture synthesis into consideration, we can 
conclude that there is no universal texture synthesizer is present. Still it remains a goal to be desire. The texture synthesis 
(Rane et al. 2002) based Inpainting perform well in approximating textures. These algorithms have difficulty in handling 
natural images as they are composed of structures in form of edges. Also they have complex interaction between structure 
and texture boundaries. In some cases, they also require the user to specify what texture to replace and the place to be 
replaced. Hence while appreciating the use of texture synthesis techniques in Inpainting, it is important to understand that 
these methods address only a small subset of Inpainting issues and these methods are not suitable for a wide variety of 
applications. 


2.2. PDE based Image Inpainting 

Partial Differential Equation (PDE) based algorithm for Image Inpainting is proposed by Bertalmio et al, (2000). This algorithm 
is the iterative algorithm. Their model is based on nonlinear partial differential equations, and is designed to imitate the 
techniques of museum artists who specialize in restoration. In particular, Bertalmio elucidated the principle that good 
inpainting algorithms should propagate sharp edges in surrounding areas into the damaged parts that need to be filled in. The 
main idea behind this algorithm is to continue geometric and photometric information that arrives at the border of the occluded 
area into area itself. This is done by propagating the information in the direction of minimal change using isophote lines. This 
algorithm will produce good results if missed regions are small one. But when the missed regions are large this algorithm will 
take so long time and it will not produce good results. Then inspired by this work, Chan and Shen, (2001) proposed the Total 
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Variational (TV) Inpainting model. This model uses Euler-Lagrange equation and anisotropic diffusion based on the strength 
of the isophotes. This model performs reasonably well for small regions and noise removal applications. But the drawback of 
this method is that this method neither connects broken edges nor greats texture patterns. The TV model then extended to 
CDD (Curvature Driven Diffusion- Chan and Shen, (2001)) model. In which it included the curvature information of the 
isophotes to handle the curved structures in a better manner. All of the above mentioned algorithms are very time consuming 
and have some problems with the damaged regions with a large size. PDE based technique has been widely used in number 
of applications such as image segmentation, restoration etc. These algorithms were focused on maintaining the structure of 
the Inpainting area. And hence these algorithms produce blurred resulting image. Another drawback of these algorithms is 
that the large textured regions are not well reproduced. 


2.3. Exemplar Based Image Inpainting 

As it was shown in above explanation, PDE based inpainting algorithms are not sufficient for faithfully reconstructing textured 
images, nor images with large missing areas. Thus, when inpainting is done with an image restoration purpose in mind, more 
complex techniques are required, as paintings are composed of both structures (i.e. primal sketches) and textures (i.e. 
regions with homogeneous patterns). Exemplar-based inpainting methods can overcome this drawback, being able to provide 
reasonably good quality results, even for large gaps, by combining the isophote driven inpainting with texture synthesis. The 
reconstructed visual quality and the reasonability of the filled image are mainly influenced by the filling order. Therefore, it’s 
our belief that better performance is reachable via developing a robust priority function. 

Exemplar based Inpainting iteratively synthesizes the unknown region i. e. target region, by the most similar patch in the 
source region. According to the filling order, the method fills structures in the missing regions using spatial information of 
neighbouring regions. This method is an efficient approach for reconstructing large target regions. Generally, an exemplar- 
based Inpainting algorithm includes the following four main steps: a) Initializing the Target Region: in which the initial missing 
areas are extracted and represented with appropriate data structures. b) Computing Filling Priorities: in this a predefined 
priority function is used to compute the filling order for all unfilled pixels say » < @Q_ in the beginning of filling iteration. c) 


Searching Example and Compositing: in which the most similar example is searched from the source region D to compose 
the given patch, say ‘¥ (of size N x N pixels) that centered on the given pixel p . d) Updating Image Information, in which the 


boundary 0Q of the target region © and the required information for computing filling priorities are updated. Numbers of 
algorithms are developed for the exemplar based image Inpainting. Such as, Bertalmio, (2003) developed a hybrid algorithm 
to combine the diffusion-based scheme and texture synthesis. This algorithm works well in recovering not only the 
geometrical structures but also the small texture regions. Then Drori, (2003) proposed a fragment-based image Inpainting 


eta based on Heat 2/gorithm that iteratively approximated, searched, and added detail by compositing adaptive fragments. The computation time 
equation (PDE), which of this algorithm is intolerable. Most of the new exemplar-based algorithms adopt the greedy strategy, so these algorithms 
propagates your pixels suffer from the common problems of the greedy algorithm, being the filling order (namely priority) is very critical. Exemplar 
from cracks borders, like based Inpainting (Wong et al. 2008) will produce good results only if the missing region consists of simple structure and 


something hot (the image) texture. And if there are not enough samples in image then it is impossible to synthesize the desired image. 

moves into a cold area (the 

crack area). 2.4. Hybrid based Image Inpainting 
The hybrid approaches combine both texture synthesis and PDE based Inpainting for completing the holes. The main idea 
behind these approaches is that it decomposed the image into two separate parts, Structure region and texture regions. The 
corresponding decomposed regions are filled by edge propagating algorithms and texture synthesis techniques. Hybrid 
inpainting technique is also called as Image Completion. It is used for filling large target (missing) regions. And also preserves 
both structure and texture in a visually plausible manner. One important direction we believe is more natural to the inpainting 
process is by structure completion through segmentation. This technique uses a two-step approach: the first stage is structure 
completion followed by texture synthesis (Rane et al. 2002). In the structure completion stage, segmentation, using the 
algorithm of, is performed based on the insouciant geometry, color and texture information on the input and then the 
partitioning boundaries are extrapolated to generate a complete segmentation for the input using tensor voting. The second 
step consists of synthesizing texture and color information in each segment, again using tensor voting. 


3. CONCLUSION 


In this comparative study, we review the existing techniques of image Inpainting. We discussed in this paper, a variety of 
different techniques of image Inpainting such as Texture synthesis based image Inpainting, PDE based image Inpainting, 
Exemplar based image Inpainting and Hybrid image Inpainting. Also we have provided a detailed explanation and description 
of the different techniques for image Inpainting which are used for filling the missing region making use of image. From this 
analysis, we also highlighted a number of shortcomings and limitations of these different techniques. 


SUMMARY OF RESEARCH 


It is observed that the PDE based image Inpainting algorithms cannot fill the large missing region and it cannot restore the texture pattern. The theoretical 
analysis proved that exemplar based Inpainting will produce good results for Inpainting the large missing region also these algorithms can inpaint both 
structure and textured image as well. But they work well only if missing region consists of simple structure and texture. 


FUTURE ISSUES 


Future study also focused on to develop such efficient algorithm that reduces computational cost and time required for Inpainting. 
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